Skip to content

Add additional checksum algorithms#6742

Merged
zoewangg merged 5 commits intofeature/master/addMoreChecksumAlgosfrom
zoewang/additionalChecksumAlgos
Feb 20, 2026
Merged

Add additional checksum algorithms#6742
zoewangg merged 5 commits intofeature/master/addMoreChecksumAlgosfrom
zoewang/additionalChecksumAlgos

Conversation

@zoewangg
Copy link
Contributor

@zoewangg zoewangg commented Feb 18, 2026

Motivation and Context

Add support for additional checksum algorithms: XXHASH64, XXHASH3, XXHASH128, SHA512, and MD5. Note that this change will be merged to a feature branch first

  • The XXHASH algorithms require the AWS CRT library.
  • MD5 is not supported for flexible checksums (httpChecksum trait). MD5 can only be used when a pre-calculated MD5 header is provided. Attempting to specify MD5 as the checksum algorithm throws IllegalArgumentException.
  • MD5 response validation is skipped in HttpChecksumUtils.shouldSkipAlgorithm() since MD5 is not a valid flexible checksum algorithm

Modifications

  • Added new checksum algorithm constants in DefaultChecksumAlgorithm: XXHASH64, XXHASH3, XXHASH128
  • Added business metrics for new checksum algorithms in BusinessMetricFeatureId and BusinessMetricsUtils
  • Changed aws-crt dependency from test scope to optional for runtime CRT checksum support
  • Updated algorithm priority list based on perf result

Large object (512MB payload, 128KB chunks):

Algorithm Time (ms) vs CRC32C Type
XXHASH3 14.5 0.65x HW-accelerated (CRT)
XXHASH128 14.5 0.65x HW-accelerated (CRT)
CRC64NVME 22.1 0.99x HW-accelerated (CRT)
CRC32C 22.4 1.0x HW-accelerated (Java intrinsic)
CRC32 27.3 1.22x Software (Java)
XXHASH64 44.1 1.97x Software (CRT)
SHA512 1030 46x Cryptographic
SHA256 1487 66x Cryptographic
SHA1 1870 84x Cryptographic

Small object (8MB payload, 128KB chunks):

Algorithm Time (ms) vs CRC32C
XXHASH3 0.21 0.61x
XXHASH128 0.22 0.63x
CRC64NVME 0.35 0.99x
CRC32C 0.35 1.0x
CRC32 0.43 1.22x
XXHASH64 0.69 1.98x
SHA512 16.1 46x
SHA256 23.3 67x
SHA1 29.3 84x

Testing

  • Added XxHashChecksumTest for XXHASH algorithm variants
  • Added XxHashNotAvailableTest to verify graceful handling when CRT is unavailable
  • Added tests for business metrics verification
  • Updated existing checksum tests to cover new algorithms

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

Checklist

  • I have read the CONTRIBUTING document
  • Local run of mvn install succeeds
  • My code follows the code style of this project
  • My change requires a change to the Javadoc documentation
  • I have updated the Javadoc documentation accordingly
  • I have added tests to cover my changes
  • All new and existing tests passed
  • I have added a changelog entry. Adding a new entry must be accomplished by running the scripts/new-change script and following the instructions. Commit the new
    file created by the script in .changes/next-release with your changes.
  • My change is to implement 1.11 parity feature and I have updated LaunchChangelog

License

  • I confirm that this pull request can be released under the Apache 2 license

@zoewangg zoewangg requested a review from a team as a code owner February 18, 2026 16:18
@zoewangg zoewangg force-pushed the zoewang/additionalChecksumAlgos branch from 20c5ebe to 1b23023 Compare February 18, 2026 21:50
@sonarqubecloud
Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
66.3% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

@zoewangg zoewangg merged commit 8b796aa into feature/master/addMoreChecksumAlgos Feb 20, 2026
27 of 30 checks passed
@zoewangg zoewangg deleted the zoewang/additionalChecksumAlgos branch February 20, 2026 03:25
@github-actions
Copy link

This pull request has been closed and the conversation has been locked. Comments on closed PRs are hard for our team to see. If you need more assistance, please open a new issue that references this one.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 20, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments